In electronic film development, conventional film is scanned electronically during development to produce a series of views of the developing image. An early scan reveals the fast developing highlight detail, while a late scan reveals slow developing shadow detail. After development, the series of views is combined into a single image in a process called stitching. In the prior art, stitching cut out the best parts of each view and merged them together. In the present invention, regression data is accumulated during development to describe a curve of density versus time of development for each pixel. After development, this regression data is used to recreate a regression curve of density versus development time for each pixel. The time at which this curve crosses a density known to give optimum grain characteristics, called the optimum density curve, is used to create the brightness for that pixel in the finished stitched image. The invention further teaches weighting regression data as a function of time and density generally following proximity to the optimum density curve.
Recording an image at different exposures and later merging the images has been practiced since the advent of photography. A technique known to photographers for overcoming the dynamic range limit of film is to make two exposures, perhaps one for the clouds and one for the shadowed foreground, and merge the two using manual printing skill in the darkroom. A similar technique is known in astrophotography where multiple exposures reveal different features of a star cluster or nebula. In a rather flashy example, Kodak developed a film in the 1950""s capable of recording the million to one brightness range of a nuclear test by making a color negative film wherein the three color layers were substituted with three monochrome layers of widely different sensitivities, each developing in color developer with a different dye color. Again, the manual skill of a darkroom printer was relied on to merge the images into one. A further example can be found in radiology where images can be made with different x-ray voltages to reveal detail in both soft and hard materials, then merging the images together. Modem color film typically uses three emulsion coatings for each color, each of a different speed. The three are merged simply by putting all three together in one film, thereby getting some benefit of a layer optimized for a particular exposure, but mixed with the grain of other layers not optimized for that particular exposure.
It was not until the advent of electronic film development, as taught in U.S. Pat. No. 5,519,510 issued to the present inventor, that there was a need to merge multiple exposure images using production-level speed and automation. In electronic film development, the merging of images is called stitching. The background of electronic film development in general and the prior art methods of stitching are now presented as a basis of understanding the background of the present invention.
Turning to FIG. 1, a scene 102, portrayed as perceived through the wide dynamic range of the human eye, has highlights 104, midtones 106, and shadows 108, with details in all areas. A camera 110 is used to project the scene onto a film inside the camera. The scene is perceived by the film to consist of points of light, each with an exposure value which may be mapped along an exposure axis 112.
The film is removed from the camera after exposure and placed in a developer. In electronic film development, an electronic camera 120 views the film by nonactinic infrared light during development. As seen after a short development time of perhaps one minute, the film 122 still has a low density for shadows 124 and midtones 126, but may optimally reveal highlights 128. As seen by the infrared camera 120, inverting for the negative of conventional film, the shadows and midtones 130 appear black, while the highlights 132 are seen more clearly than at any later time in development.
Doubling development time to two minutes, the midtones 140 have progressed to an optimum density while the highlights 142 may already be overdeveloped and the shadows 144 may still be too low in density to reveal a clear image. The film 146 would appear to have good midtone detail 148, but the highlights 150 are already white, while the shadows 152 are still black.
Doubling development time again to a total of four minutes, the shadows have now reached an optimum density, but the other exposures are overdeveloped such that in image 162 they may appear white with little detail.
For each exposure, there is an optimum density of development to reveal the clearest image. Clarity may be defined technically as the best signal to noise ratio, where signal is the incremental change in density with exposure, and noise is the RMS deviation in density across a region that has received uniform exposure, by convention scanned with a 24 micron aperture. For example at one minute of development time, the midtones 126 typically have too low a density, or are too dark, to have enough of a signal level to reveal detail through the noise of the film and capture system. On the other hand, at four minutes the midtones 164 are xe2x80x9cwashed outxe2x80x9d, such that not only is their contrast, or image signal strength, too low, but the graininess of an overdeveloped silver halide emulsion gives a high noise. There exists a development time in between these extremes, two minutes in this example, wherein the midtones 140 have developed to an optimum density that yields the best signal to noise ratio, or image clarity, for that particular exposure value. In this example, the shadows reach optimum clarity at four minutes of development 160, and the highlights reach optimum clarity at one minute of development 128. In general, the optimum density will be different for different exposures, as in this example wherein the shadows 160 reveal best clarity at a lower density than the highlights 128.
After the final capture of the image on the film at four minutes, electronic film development has captured optimum images for shadows, midtones, and highlights albeit at different development times. These optimum images must be combined to form a single image with clarity throughout approximating the original scene as seen by the wide dynamic range of the human eye. The process of combining these different parts of the image is called stitching. The prior art conceived this in the classic sense of merging multiple films in a darkroom by cutting out the shadows, the midtones, and the highlights, lightening and darkening each so the boundaries between regions aligned, then stitching these multiple images together into one.
The advantage of electronic film development is now more easily understood. In conventional development the film must be stopped and fixed at a selected development time, such as the two minute development time of this example. The detail of the highlights revealed at one minute is lost in total darkness as conventional development proceeds. Likewise, the detail that might have been revealed at four minutes never had the chance to be born in conventional development. Electronic film development turns conventional film into a xe2x80x9cuniversalxe2x80x9d film that can be used at a wide range of exposure indexes, including very high exposure indexes not currently practical.
In FIG. 1, the section of the density curve around the optimally developed shadows 160 is copied as segment 170. Next, the density curve around the optimally developed midtones 140 is raised on a base value, or pedestal 172, and copied next to curve 170 as curve 174. The height of the pedestal 172 is adjusted so the two curves 170 and 174 align. Similarly, the curve around the optimally developed highlights 128 is adjusted and raised on pedestal 176 to produce curve 178. The process works in theory, but in practice, development nonuniformities across the image and other spatially dependent nonlinearities made the curves difficult to match across an entire image so that the stitched image usually displayed contours at the edges of stitching regions. Obviously, an improved method of stitching was needed to realize the full benefits of electronic film development.
Often in electronic film development there are more than three exposures made of the film. For example, an area array camera may view the film continuously, generating hundreds of exposures. In the prior art these needed to be combined into a limited number of images to conserve memory during the capture process. For example, in FIG. 2 the exposures made at one-half and one minute, exposures 202 and 204, respectively, could be added with function block 206 to produce a single short development image 208. Similarly, various exposures at other development times could be added to yield a middle development image 210 and a late development image 212. These images would then be aligned, cut, and pasted together at function block 220 to yield the finished image 222. A problem is immediately seen if the times of capture vary, making it necessary to adjust the densities in each of the intermediate images by known time deviations. In the past the adjustments were based on estimations of development speed, and were not found to be reliable. In addition, there were difficulties if some of the capture times were missing entirely because, perhaps, a non real time operating system did not release computer resources exactly when needed.
Electronic film development held the promise of higher speed universal film that would work in conventional cameras. This higher speed and wider range film would enable families to record their lives beautifully in the natural light of real life, without typical problems caused by contrast light or dependence on a cold and harsh flash. However, the prior art implementations of electronic film development were plagued with problems in stitching the multiple exposure images together. Obviously, an improved stitching method is an important advance to the art.
The primary object of the invention is to merge images of differing densities into a single image which is free from the artifacts encountered in the prior art.
A related object is to merge images of differing densities free of edge contouring.
A further object is to merge images of differing densities with reduced effect from nonimage noise.
A further object is to merge images of differing densities while compensating for a shift in a density-affecting parameter, such as time.
Another object is to recover missed images in a series of images of differing densities that are to be merged.
In the present invention, a series of images are captured electronically from a developing film, each tagged with the time of capture. For each pixel of each image at each time, regression parameters are calculated, such as density times time, or density times time squared. These parameters for each time are summed into parameter accumulating arrays. As a refinement, the parameters can be weighted prior to summing by a factor sensitive to the reliability of each sample. Following film development, the regression statistics are not necessarily viewable images, rather they describe in abstract mathematical terms smooth continuous lines for each pixel that pass through the actual sampled densities for each pixel. These mathematically described smooth lines allow the development to be recreated mathematically in order to find the nonquantized time at which the density of each pixel is predicted to have attained its optimum density. A gamma correction function of this nonquantized time for each pixel is then output as the brightness for that pixel.